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1 . A method for evaluating a computer learning signal processing engine, comprising: 
identifying a first group of signal sets, each signal set of the first group having an 

associated range of values for a variable corresponding to the first group, the variable being 
one of a plurality of variables having values characterizing multiple signals to be processed; 

calculating an accuracy score for each signal set of the first group using the signal 
processing engine to be evaluated; 

applying weight factors to the accuracy scores for the first group signal sets, each 
weight factor representing a relative importance of one of the associated ranges of values for 
the first variable; 

summing weighted accuracy scores for the first group of signal sets to yield a first 
summed accuracy score; 

identifying additional groups of signal sets, each group having a corresponding 
variable of the plurality of variables, each signal set of a group having an associated range of 
values for the corresponding variable; 

calculating accuracy scores for each signal set of each additional group using the 
signal processing engine to be evaluated; 

applying weight factors to the accuracy scores for the signal sets of the additional 
groups, the weight factors within each of the additional groups representing a relative 
importance of associated ranges of values for the variable corresponding to the group; 

summing the weighted accuracy scores within each of the additional groups to yield 
additional summed accuracy scores; and 

further combining the summed accuracy scores. 

2. The method of claim 1, wherein at least some of the associated ranges of values are 
single values. 

3. The method of claim 1, wherein said further combining the summed accuracy scores 
comprises: 
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applying weight factors to a plurality of the summed accuracy scores, each of 
the weight factors applied to the plurality of summed accuracy scores representing a relative 
importance of the variable corresponding to the group from which a different one of the 
plurality of summed accuracy scores was derived, and 

su mmin g the plurality of weighted summed accuracy scores to yield a 
subsequent sum. 

4. The method of claim 3, wherein said further combining the summed accuracy scores 
comprises: 

applying additional weight factors to the subsequent sum and to at least one of 
the summed accuracy scores, the additional weight factors representing the relative 
importance of at least one variable corresponding to groups from which the subsequent sum 
was derived and of the variable corresponding to the group from which the at least one 
summed accuracy score was derived, and 

summing the additionally weighted subsequent sum and the additionally 
weighted at least one summed accuracy score. 

5. The method of claim 3, further comprising: 

identifying an additional variable having values characterizing multiple signals to be 
processed; 

identifying another group of signal sets, each signal set of the group having an 
associated range of values for the additional variable; 

calculating an accuracy score for each signal set of the additional variable group using 
the signal processing engine to be evaluated; 

applying weight factors to the accuracy scores for the additional group signal sets, 
each weight factor representing a relative importance of one of the associated ranges of 
values for the additional variable; 

summing weighted accuracy scores for additional group signal sets to yield an 
additional summed accuracy score; 
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applying weight factors to the plurality of the summed accuracy scores and to the 
additional summed accuracy score, each of the weight factors representing a relative 
importance of the variable corresponding to the group from which a different one of the 
plurality of summed accuracy scores was derived or of the additional variable, and 

summing the plurality of weighted summed accuracy scores and the weighted 
additional accuracy score. 

6. The method of claim 1 , further comprising 
selecting a variable of the plurality of variables; 

selecting sub-variables, each sub-variable having a range of values for a value of the 
selected variable; 

calculating an accuracy score for signal sets in groups of signal sets corresponding to 
the sub-variables, each of the signal sets within a group corresponding to a range of values 
for the corresponding sub-variable; 

applying weights to the accuracy scores for the signal sets in each sub-variable group 
and summing the weighted scores within each of said groups to yield sub-variable accuracy 
scores; and 

applying weights to the sub-variable accuracy scores and summing the weighted sub- 
variable accuracy scores. 

7. The method of claim 1, wherein: 

at least one variable is a source variable having values characterizing a source 
of a signal to be processed, 

at least one variable is a context variable having values characterizing the 
context of a signal to be processed, and 

at least one variable is a physical variable having values characterizing 
physical attributes of a signal to be processed. 
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8. The method of claim 7, wherein at least one variable has values characterizing a user 
scenario in which a signal was generated, the user scenario values including at least one of a 
software application and an operation performed within a software application. 

9. The method of claim 8, wherein: 

the signal processing engine to be evaluated comprises handwriting 
recognition software, 

the signals to be processed comprise handwriting samples, 

values of the source variable comprise demographic data regarding users 
creating handwriting samples, 

values of the context variable comprise data regarding the context of 
handwriting samples, and 

values of the physical variable comprise at least one of data regarding the 
scaling of a handwriting sample, data regarding the relative angle of components of a 
handwriting sample and data regarding the spacing between components of a handwriting 
sample. 

10. The method of claim 9, wherein: 

the groups of signal sets comprise: 

groups of demographic signal sets, each demographic signal set in a 
group having an associated range of values for the source variable and the same value for the 
context variable, 

groups of scaling signal sets, each scaling signal set in a group having 
an associated range of values for the scaling of a handwriting sample and the same value for 
the context variable, and 

groups of angle signal sets, each angle signal set in a group having an 
associated range of values for at least one angle of a component of a handwriting sample and 
the same value for the context variable, 
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at least the summed accuracy scores for demographic signal set, scaling signal 
set and angle signal set groups of signals having a common context variable value are 
weighted and summed to yield context accuracy scores, 

context accuracy scores are weighted and summed to yield a combined context 
accuracy score, 

the combined context accuracy score and at least one other summed accuracy 
score are weighted and summed to yield a word accuracy score, and 

at least the word accuracy score and a summed accuracy score for a group 
corresponding to the user scenario variable are weighted and summed to yield an overall 
accuracy score. 

1 1 . The method of claim 8, wherein: 

the signal processing engine to be evaluated comprises speech recognition 

software, 

the signals to be processed comprise speech samples, 

values of the source variable comprise demographic data regarding users 
creating speech samples, 

values of the context variable comprise data regarding the context of speech 

samples, and 

values of the physical variable comprise at least one of data regarding the 
scaling of a speech sample and data regarding the spacing between components of a speech 
sample. 

12. The method of claim 1 1, wherein: 

the groups of signal sets comprise: 

groups of demographic signal sets, each demographic signal set in a 
group having an associated range of values for the source variable and the same value for the 
context variable, and 
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groups of scaling signal sets, each scaling signal set in a group having 
an associated range of values for the scaling of a speech sample and the same value for the 
context variable, 

at least the summed accuracy scores for demographic signal set and scaling 
signal set groups of signals having a common context variable value are weighted and 
summed to yield context accuracy scores, 

context accuracy scores are weighted and summed to yield a combined context 
accuracy score, 

the combined context accuracy score and at least one other summed accuracy 
score are weighted and summed to yield a word accuracy score, and 

at least the word accuracy score and a summed accuracy score for a group 
corresponding to the user scenario variable are weighted and summed to yield an overall 
accuracy score. 

13. The method of claim 1, further comprising applying a transforming function to a sum. 

14. The method of claim 13, wherein the transform function comprises: 

outputting the sum if the sum is above a threshold, and 

outputting number having a large absolute value if the sum is not above the 

threshold. 

1 5 . The method of claim 1 , further comprising: 
setting accuracy scores for signal sets to equal 1; 
applying confidence scores to weight factors; and 
calculating an overall confidence score. 

16. A computer-readable medium having stored thereon data representing sequences of 
instructions which, when executed by a processor, cause the processor to perform steps 
comprising: 
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identifying a first group of signal sets, each signal set of the first group having an 
associated range of values for a variable corresponding to the first group, the variable being 
one of a plurality of variables having values characterizing multiple signals to be processed; 

calculating an accuracy score for each signal set of the first group using a signal 
processing engine to be evaluated; 

applying weight factors to the accuracy scores for the first group signal sets, each 
weight factor representing a relative importance of one of the associated ranges of values for 
the first variable; 

summing weighted accuracy scores for first group signal sets to yield a first summed 
accuracy score; 

identifying additional groups of signal sets, each group having a corresponding 
variable of the plurality, each signal set of a group having an associated range of values for 
the corresponding variable; 

calculating accuracy scores for each signal set of each additional group using the 
signal processing engine to be evaluated; 

applying weight factors to the accuracy scores for the signal sets of the additional 
groups, the weight factors within each of the additional groups representing a relative 
importance of associated ranges of values for the variable corresponding to the group; 

summing the weighted accuracy scores within each of the additional groups to yield 
additional summed accuracy scores; and 

further combining the summed accuracy scores. 

17. The computer-readable medium of claim 16, wherein at least some of the associated 
ranges of values are single values. 

18. The computer-readable medium of claim 16, wherein said further combining the 
summed accuracy scores comprises: 

applying weight factors to a plurality of the summed accuracy scores, each of 
the weight factors applied to the plurality of summed accuracy scores representing a relative 



-37- 



Patent Application Atty. Docket No. 003797.00735 

importance of the variable corresponding to the group from which a different one of the 
plurality of summed accuracy scores was derived, and 

summing the plurality of weighted summed accuracy scores to yield a 
subsequent sum. 

19. The computer-readable medium of claim 18, wherein said further combining the 
summed accuracy scores comprises: 

applying, additional weight factors to the subsequent sum and to at least one of 
the summed accuracy scores, the additional weight factors representing the relative 
importance of at least one variable corresponding to groups from which the subsequent sum 
was derived and of the variable corresponding to the group from which the at least one 
summed accuracy score was derived, and 

summing the additionally weighted subsequent sum and the additionally 
weighted at least one summed accuracy score. 

20. The computer-readable medium of claim 18, comprising additional data representing 
sequences of instructions which, when executed by a processor, cause the processor to 
perform additional steps comprising: 

identifying an additional variable having values characterizing multiple signals to be 
processed; 

identifying another group of signal sets, each signal set of the group having an 
associated range of values for the additional variable; 

calculating an accuracy score for each signal set of the additional variable group using 
the signal processing engine to be evaluated; 

applying weight factors to the accuracy scores for the additional group signal sets, 
each weight factor representing a relative importance of one of the associated ranges of 
values for the additional variable; 

summing weighted accuracy scores for additional group signal sets to yield an 
additional summed accuracy score; 
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applying weight factors to the plurality of the summed accuracy scores and to the 
additional summed accuracy score, each of the weight factors representing a relative 
importance of the variable corresponding to the group from which a different one of the 
plurality of summed accuracy scores was derived or of the additional variable, and 

summing the plurality of weighted summed accuracy scores and the weighted 
additional accuracy score. 

21. The computer-readable medium of claim 16, comprising additional data representing 
sequences of instructions which, when executed by a processor, cause the processor to 
perform additional steps comprising: 

selecting a variable of the plurality; 

selecting sub- variables, each sub-variable having a range of values for a value of the 
selected variable; 

calculating an accuracy score for signal sets in groups of signal sets corresponding to 
the sub-variables, each of the signal sets within a group corresponding to a range of values 
for the corresponding sub-variable; 

applying weights to the accuracy scores for the signal sets in each sub-variable group 
and summing the weighted scores within each of said groups to yield sub-variable accuracy 
scores; and 

applying weights to the sub-variable accuracy scores and s ummin g the weighted sub- 
variable accuracy scores. 

22. The computer-readable medium of claim 16, wherein: 

at least one variable is a source variable having values characterizing a source 
of a signal to be processed, 

at least one variable is a context variable having values characterizing the 
context of a signal to be processed, and 

at least one variable is a physical variable having values characterizing 
physical attributes of a signal to be processed. 
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23. The computer-readable medium of claim 22, wherein at least one variable has values 
characterizing a user scenario in which a signal was generated, the user scenario values 
including at least one of a software application and an operation performed within a software 
application. 

24. The computer-readable medium of claim 23, wherein: 

the signal processing engine to be evaluated comprises handwriting 
recognition software, 

the signals to be processed comprise handwriting samples, 

values of the source variable comprise demographic data regarding users 
creating handwriting samples, 

values of the context variable comprise data regarding the context of 
handwriting samples, and 

values of the physical variable comprise at least one of data regarding the 
scaling of a handwriting sample, data regarding the relative angle of components of a 
handwriting sample and data regarding the spacing between components of a handwriting 
sample. 

25. The computer-readable medium of claim 24, wherein: 

the groups of signal sets comprise: 

groups of demographic signal sets, each demographic signal set in a 
group having an associated range of values for the source variable and the same value for the 
context variable, 

groups of scaling signal sets, each scaling signal set in a group having 
an associated range of values for the scaling of a handwriting sample and the same value for 
the context variable, and 

groups of angle signal sets, each angle signal set in a group having an 
associated range of values for at least one angle of a component of a handwriting sample and 
the same value for the context variable, 
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at least the summed accuracy scores for demographic signal set, scaling signal 
set and angle signal set groups of signals having a common context variable value are 
weighted and summed to yield context accuracy scores, 

context accuracy scores are weighted and summed to yield a combined context 
accuracy score, 

the combined context accuracy score and at least one other summed accuracy 
score are weighted and summed to yield a word accuracy score, and 

at least the word accuracy score and a summed accuracy score for a group 
corresponding to the user scenario variable are weighted and summed to yield an overall 
accuracy score. 

26. The computer-readable medium of claim 23, wherein: 

the signal processing engine to be evaluated comprises speech recognition 

software, 

the signals to be processed comprise speech samples, 

values of the source variable comprise demographic data regarding users 
creating speech samples, 

values of the context variable comprise data regarding the context of speech 

samples, and 

values of the physical variable comprise at least one of data regarding the 
scaling of a speech sample and data regarding the spacing between components of a speech 
sample. 

27. The computer-readable medium of claim 26, wherein: 

the groups of signal sets comprise: 

groups of demographic signal sets, each demographic signal set in a 
group having an associated range of values for the source variable and the same value for the 
context variable, and 
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groups of scaling signal sets, each scaling signal set in a group having 
an associated range of values for the scaling of a speech sample and the same value for the 
context variable, 

at least the summed accuracy scores for demographic signal set and scaling 
signal set groups of signals having a common context variable value are weighted and 
summed to yield context accuracy scores, 

context accuracy scores are weighted and summed to yield a combined context 
accuracy score, 

the combined context accuracy score and at least one other summed accuracy 
score are weighted and summed to yield a word accuracy score, and 

at least the word accuracy score and a summed accuracy score for a group 
corresponding to the user scenario variable are weighted and summed to yield an overall 
accuracy score. 

28. The computer-readable medium of claim 16, comprising additional data representing 
sequences of instructions which, when executed by a processor, cause the processor to 
perform additional steps comprising applying a transforming function to a sum. 

29. The computer-readable medium of claim 28, wherein the transform function 
comprises: 

outputting the sum if the sum is above a threshold, and 

outputting number having a large absolute value if the sum is not above the 

threshold. 

30. The computer-readable medium of claim 16, comprising additional data representing 
sequences of instructions which, when executed by a processor, cause the processor to 
perform additional steps comprising: 

setting accuracy scores for signal sets to equal 1; 
applying confidence scores to weight factors; and 
calculating an overall confidence score. 
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